Today's large language models (LLMs) are capable of supporting multilingual scenarios, allowing users to interact with LLMs in their native languages. When LLMs respond to subjective questions posed by users, they are expected to align with the views of specific demographic groups or historical periods, shaped by the language in which the user interacts with the model. Existing studies mainly focus on researching the opinions represented by LLMs among demographic groups in the United States or a few countries, lacking worldwide country samples and studies on human opinions in different historical periods, as well as lacking discussion on using language to steer LLMs. Moreover, they also overlook the potential influence of prompt language on the alignment of LLMs' opinions. In this study, our goal is to fill these gaps. To this end, we create an evaluation framework based on the World Values Survey (WVS) to systematically assess the alignment of LLMs with human opinions across different countries, languages, and historical periods around the world. We find that LLMs appropriately or over-align the opinions with only a few countries while under-aligning the opinions with most countries. Furthermore, changing the language of the prompt to match the language used in the questionnaire can effectively steer LLMs to align with the opinions of the corresponding country more effectively than existing steering methods. At the same time, LLMs are more aligned with the opinions of the contemporary population. To our knowledge, our study is the first comprehensive investigation of the topic of opinion alignment in LLMs across global, language, and temporal dimensions. Our code and data are publicly available at https://github.com/ku-nlp/global-opinion-alignment and https://github.com/nlply/global-opinion-alignment.

As Large Language Models (LLMs) increasingly appear in social science research (e.g., economics and marketing), it becomes crucial to assess how well these models replicate human behavior. In this work, using hypothesis testing, we present a quantitative framework to assess the misalignment between LLM-simulated and actual human behaviors in multiple-choice survey settings. This framework allows us to determine in a principled way whether a specific language model can effectively simulate human opinions, decision-making, and general behaviors represented through multiple-choice options. We applied this framework to a popular language model for simulating people's opinions in various public surveys and found that this model is ill-suited for simulating the tested sub-populations (e.g., across different races, ages, and incomes) for contentious questions. This raises questions about the alignment of this language model with the tested populations, highlighting the need for new practices in using LLMs for social science studies beyond naive simulations of human subjects.

The increased adoption of Large Language Models (LLMs) and their potential to shape public opinion have sparked interest in assessing these models' political leanings. Building on previous research that compared LLMs and human opinions and observed political bias in system responses, we take a step further to investigate the underlying causes of such biases by empirically examining how the values and biases embedded in training corpora shape model outputs. Specifically, we propose a method to quantitatively evaluate political leanings embedded in the large pretraining corpora. Subsequently we investigate to whom are the LLMs' political leanings more aligned with, their pretrainig corpora or the surveyed human opinions. As a case study, we focus on probing the political leanings of LLMs in 32 U.S. Supreme Court cases, addressing contentious topics such as abortion and voting rights. Our findings reveal that LLMs strongly reflect the political leanings in their training data, and no strong correlation is observed with their alignment to human opinions as expressed in surveys. These results underscore the importance of responsible curation of training data and the need for robust evaluation metrics to ensure LLMs' alignment with human-centered values.

Recent years have seen the rise of large language models (LLMs), where practitioners use task-specific prompts; this was shown to be effective for a variety of tasks. However, when applied to semantic textual similarity (STS) and natural language inference (NLI), the effectiveness of LLMs turns out to be limited by low-resource domain accuracy, model overconfidence, and difficulty to capture the disagreements between human judgements. With this in mind, here we try to rethink STS and NLI in the era of LLMs. We first evaluate the performance of STS and NLI in the clinical/biomedical domain, and then we assess LLMs' predictive confidence and their capability of capturing collective human opinions. We find that these old problems are still to be properly addressed in the era of LLMs.

Were you unable to attend Transform 2022? Check out all of the summit sessions in our on-demand library now! The training process for artificial intelligence (AI) algorithms is designed to be largely automated innately. There are often thousands, millions or even billions of data points and the algorithms must process all of them to search for patterns. In some cases, though, AI scientists are finding that the algorithms can be made more accurate and efficient if humans are consulted, at least occasionally, during the training.

Were you unable to attend Transform 2022? Check out all of the summit sessions in our on-demand library now! The training process for artificial intelligence (AI) algorithms is designed to be largely automated innately. There are often thousands, millions or even billions of data points and the algorithms must process all of them to search for patterns. In some cases, though, AI scientists are finding that the algorithms can be made more accurate and efficient if humans are consulted, at least occasionally, during the training.

Despite the subjective nature of many NLP tasks, most NLU evaluations have focused on using the majority label with presumably high agreement as the ground truth. Less attention has been paid to the distribution of human opinions. We collect ChaosNLI, a dataset with a total of 464,500 annotations to study Collective HumAn OpinionS in oft-used NLI evaluation sets. This dataset is created by collecting 100 annotations per example for 3,113 examples in SNLI and MNLI and 1,532 examples in Abductive-NLI. Analysis reveals that: (1) high human disagreement exists in a noticeable amount of examples in these datasets; (2) the state-of-the-art models lack the ability to recover the distribution over human labels; (3) models achieve near-perfect accuracy on the subset of data with a high level of human agreement, whereas they can barely beat a random guess on the data with low levels of human agreement, which compose most of the common errors made by state-of-the-art models on the evaluation sets. This questions the validity of improving model performance on old metrics for the low-agreement part of evaluation datasets. Hence, we argue for a detailed examination of human agreement in future data collection efforts, and evaluating model outputs against the distribution over collective human opinions. The ChaosNLI dataset and experimental scripts are available at https://github.com/easonnie/ChaosNLI

For over a year, I worked as a beauty editor, writing and researching about the products, trends, and people that make us want to look a certain way. And as research for many of the stories I wrote, I consulted with dermatologists, plastic surgeons, makeup artists, aestheticians, and more trying to answer a simple question--how can I make myself more conventionally attractive? "Beauty is confidence," they'd always say, prefacing the real answer. Inevitably, these experts would eventually tell me that you feel more confident, and thus more beautiful, when you look blemish- and wrinkle-free. Naturally, the problem here is the premise.

The second beauty contest, where humans are judged by the robots completes with over six thousand images evaluated by the five robot judges. In addition to the panel of judges from the first contest, Beauty.AI 2.0 featured three new robot judges including: "Average Face" built on the hypothesis that the closer the face is to the average face within the ethnic group, the more attractive it is "AntiAgeist" evaluating the difference between the predicted and actual chronological age "PIMPL" evaluating the number and distribution of pimples and other dark spots (but not freckles) The results were sent to the individual participants via secure link and winners were announced at http://winners2.beauty.ai/#win . The results were surprising, since the consensus scores provided by the robot jury disagreed with human opinion. Tens of participants responded with angry emails criticizing the winners selected by the robot jury. Statements including "what is your "robot" worth??? One walk through a shopping-mall and I will discover more attractive people vs. that ones "won" your Beauty Contest", "If this is how I will be judged in the future, I don't want to see it", "You need human opinion" were among the most pleasant ones with rare positive comments including "this contest is a confidence booster!".